feat: set evidence-based memory requests for backend and mgmt-agent - #6324
Merged
openshift-merge-bot[bot] merged 1 commit intoAug 3, 2026
Merged
Conversation
openshift-ci
Bot
requested review from
Gerd Oberlechner (geoberle) and
Mariusz Mazur (mmazur)
July 30, 2026 04:27
Contributor
There was a problem hiding this comment.
Pull request overview
Adjusts Kubernetes memory requests for the backend and mgmt-agent services to better reflect observed production usage, and standardizes mgmt-agent resource configuration by sourcing requests from config.yaml rather than hardcoded chart defaults.
Changes:
- Increase backend memory request default from
500Mito1Gi. - Increase mgmt-agent memory request from
64Mito768Mi. - Wire mgmt-agent request values through
config.yamland regenerate rendered configs / Helm fixtures.
Reviewed changes
Copilot reviewed 12 out of 12 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| mgmt-agent/zz_fixture_TestHelmTemplate_dev_westus3_mgmt_1_mgmt_agent.yaml | Updates Helm fixture to reflect new mgmt-agent memory request (768Mi). |
| mgmt-agent/values.yaml | Sources mgmt-agent resource requests from config.yaml instead of hardcoded values. |
| config/rendered/dev/pers/westus3.yaml | Rendered config updated for backend (1Gi) and mgmt-agent resources. |
| config/rendered/dev/perf/westus3.yaml | Rendered config updated for backend (1Gi) and mgmt-agent resources. |
| config/rendered/dev/dev/westus3.yaml | Rendered config updated for backend (1Gi) and mgmt-agent resources. |
| config/rendered/dev/cspr/westus3.yaml | Rendered config updated for backend (1Gi) and mgmt-agent resources. |
| config/rendered/dev/ci01/centralus.yaml | Rendered config updated for backend (1Gi) and mgmt-agent resources. |
| config/rendered/dev/ci00/centralus.yaml | Rendered config updated for backend (1Gi) and mgmt-agent resources. |
| config/config.yaml | Updates defaults: backend request to 1Gi; adds mgmt-agent resources defaults. |
| backend/zz_fixture_TestHelmTemplate_dev_westus3_svc_1_aro_hcp_backend_dev.yaml | Updates backend Helm fixture to reflect new memory request (1Gi). |
| backend/testdata/zz_fixture_TestHelmTemplate_backend_mi_mock_and_arm_perms_mgr_identities_unset.yaml | Updates backend test fixture to reflect new memory request (1Gi). |
| backend/testdata/zz_fixture_TestHelmTemplate_backend_clstr_scoped_identities_role_set_name_public.yaml | Updates backend test fixture to reflect new memory request (1Gi). |
Wenqi He (weherdh)
force-pushed
the
aroslsre-1033-1654-memory-requests
branch
from
July 30, 2026 04:59
4de15ab to
032a085
Compare
Backend: 500Mi → 1Gi (actual steady-state: 750MB–1GB in uksouth) mgmt-agent: 64Mi → 768Mi (actual steady-state: 256–512MB, peak <768MB) Values determined by querying container_memory_working_set_bytes across 3 prod regions for 7 days using alert-tester. Formula: peak × 1.25 margin. Ref: AROSLSRE-1033, AROSLSRE-1654
Wenqi He (weherdh)
force-pushed
the
aroslsre-1033-1654-memory-requests
branch
from
July 31, 2026 00:55
032a085 to
e22028c
Compare
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 14 out of 14 changed files in this pull request and generated no new comments.
Comments suppressed due to low confidence (2)
mgmt-agent/deploy/templates/deployment.yaml:73
- The PR description says “Priority class changes are tracked separately and will follow”, but this change adds
priorityClassName: service-lifecycle-criticalto the mgmt-agent Deployment. Please either remove this priority class change from this PR (and regenerate the helm fixtures) or update the PR description/ticket scope to explicitly include it.
serviceAccountName: {{ .Values.serviceAccount.name }}
priorityClassName: service-lifecycle-critical
terminationGracePeriodSeconds: 10
backend/deploy/templates/backend.deployment.yaml:44
- The PR description notes priority class changes are being tracked separately, but this PR adds
priorityClassName: service-lifecycle-criticalto the backend Deployment. Please either drop the priority class change from this PR (and update the generated fixtures) or update the PR description so the scope matches the code.
app: aro-hcp-backend
serviceAccountName: '{{ .Values.serviceAccount.name }}'
priorityClassName: service-lifecycle-critical
volumes:
Rael Garcia (raelga)
approved these changes
Aug 3, 2026
Rael Garcia (raelga)
left a comment
Rael Garcia (raelga)
left a comment
Collaborator
There was a problem hiding this comment.
/lgtm
/approve
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: raelga, weherdh The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
Contributor
Contributor
Contributor
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
https://redhat.atlassian.net/browse/AROSLSRE-1033
https://redhat.atlassian.net/browse/AROSLSRE-1654
What
Set evidence-based memory requests for two services that are currently under-provisioned:
500Mi→1Gi(was running at 1.5–3.25x its request in uksouth)64Mi→768Mi(was running at 8–10x its request — 64Mi was never a real sizing)Also wires mgmt-agent's resource values through
config.yamlinstead of hardcoded defaults invalues.yaml.Why
Both services have memory requests that don't reflect actual production usage:
ServiceMemoryDriftalert (PR feat: add generic memory drift and trend alerts for ARO-HCP services #6011).Evidence gathered by running
alert-testeragainst prod Grafana (container_memory_working_set_bytes) across uksouth, eastus2, and australiaeast for July 21–28.Testing
make materializepasses — rendered configs and helm fixtures regeneratedServiceMemoryDriftalert (PR feat: add generic memory drift and trend alerts for ARO-HCP services #6011) will validate these values once deployed — right-sized requests should produce 0 firingsSpecial notes for your reviewer
values.yamlnow references{{ .mgmtAgent.k8s.resources.requests.memory }}instead of a hardcoded64Mi, consistent with how other services (backend, kube-applier) wire their resource values.PR Checklist